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(54) Recognition of handwriting in mobile station 



(57) The invention relates to improving the functions 
and applicability of a mobile station (201), (301), (401). 
Advantageous embodiments of the invention include a 
method in which graphic information may be input to a 
terminal equipment via a touch-sensitive writing pad and 
at the same time presented on a display in parallel with 
the independent execution of other functions, and a 
portable terminal equipment according to the method, 
which comprises a voice communicator (for text and file 



storage and transmission functions) and a display for 
presenting visual information as well as a graphic writing 
pad for entering input data in graphic format to the ter- 
minal equipment. In a communicator according to an 
embodiment of the invention a video terminal may be 
used in the information input simultaneously as a graph- 
ic writing pad and a display of the video terminal. The 
invention enables deaf and dumb people to communi- 
cate and take part in a video conference, for example. 
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Description 

[0001] The invention relates to the properties and ap- 
plicability of a mobile station. 

[0002] A video conference or communication with e. 
g. deaf and dumb people requires the use of multimedia 
in which participants may send information (not just pic- 
tures, sound and video but also text-based information) 
to other participants and, on the other hand, receive and 
utilize information sent by other participants. 
[0003] In addition to voice properties a modern port- 
able phone may also have functions enabling it to trans- 
mit and receive text messages or multimedia messages , 
for example. Moreover, it is known that it is possible to 
transmit, receive and present a video image in digital 
format on a mobile station using a camera and a display 
attached to the mobile station. 
[0004] Existing multimedia properties could be uti- 
lized e.g. in a video conference in which the participants 
communicate through a so-called video terminal. In the 
future it is advantageous that the characteristics of mo- 
bile stations and other terminal equipment can be made 
even more versatile. 

[0005] Accordingly, it is an object of the invention to 
provide a versatile means of communication and a flex- 
ible user interface with wide applicability. In addition, it 
Is an object of the invention to improve the characteris- 
tics of multimedia communicators. This can be achieved 
by providing the display or corresponding writing sur- 
face in a mobile station with text recognition capability. 
[0006] The objects of the invention are achieved by 
using the display or a portion of the display of a mobile 
video phone or corresponding video communicator 
serving as a portable multimedia terminal or a separate 
writing surface as a touch-sensitive writing pad for the 
recognition and storing of handwritten text. To the stor- 
age it is possible to connect the use of a handwriting 
recognition algorithm for real-time updating and trans- 
mission of notes and text. 

[0007] One embodiment of the invention is a method 
belonging to the realm of multimedia technology in 
which the touch-sensitive display of a mobile phone and 
a writing means, say an ordinary pencil, are used for 
real-time storing of handwritten text while at the same 
time the display may output other information such as a 
received video image, for example. At the same time the 
text may be converted into character-based format us- 
ing a special recognition algorithm as well as be dis- 
played on a video screen or on the apparatuses of the 
other participants in the video ; call. Texts may be sent 
and stored in a pixel- or character-based file for later 
use, say to be transmitted to a mobile communication 
or other network, and/or be used as parts of graphic im- 
ages and text on the display. In addition to handwritten 
character recognition the method may be used to rec- 
ognize the identity of the writer. 
[0008] The invention will now be described in more 
detail with reference to the advantageous embodiments 
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presented by way of example and to the accompanying 
drawings wherein 

Fig. 1 illustrates the properties of a mobile multime- 
dia communicator with respect to the inven- 
tion, 

Fig. 2 illustrates the principle and functions of the in- 
vention when the embodiment of the invention 
is a mobile multimedia commu nicator or a cor- 
responding portable terminal, 

Fig. 3 illustrates the tasks of functions control ac- 
cording to the invention in a mobile station, 

Fig. 4 shows a functional block diagram of an appa- 
ratus according to an embodiment of the in- 
vention. 



20 [0009] Electronic recognition of handwritten charac- 
ters is known as such. Handwritten text can be recog- 
nized into characters using e.g. 3Com's Palm Pilot de- 
vices. Palm Pilot is a registered trademark owned by 
3Com. It is known as such to add handwritten text rec- 

25 ognition capability to an ordinary mobile phone (say, 
Qualcomm), as is disclosed e.g. in EP782357 A2. For 
character recognition has been proposed a motion de- 
tector in a portable phone system built in the form of a 
pen, with which actions directed to the text can be con- 

30 trolled by means of a special selector (GB 2329300A 
and EP0907278 A). 

[0010] Moreover, proposals are known for telephone 
apparatuses without a handwriting recognition algo- 
rithm, in which apparatuses Asian characters to be 

35 transmitted either directly or through a cache memory 
are written by hand on a suitable sensitive display in a 
phone (EP782357 A2), whereby it is proposed that the 
display could be thermally, electrostatically, magnetical- 
ly or optically sensitive or pressure sensitive so as to 

40 react to the action of a writing means. 

[0011] Additionally, proposals are known for electric 
methods to store and process handwritten information 
in a compressed pixel-based image. The image is not 
encoded to a format understood by a computer in the 

45 terminal itself, but the encoding is carried out later in a 
separate external unit. The handwritten figures are de- 
tected in a writing block, by a pressure-sensitive mod- 
ule, as pixel-based graphics the information in which 
can be forwarded for encoding in a computer via a cable 

so or RAM card (DE 4428836 A). 

[0012] The present invention uses a touch-sensitive 
display or a portion of such a display in the storage and 
display of handwritten alphanumeric text in conjunction 
with an algorithm that recognizes handwritten text to be 

55 transmitted and/or stored in character-based format. 
The invention enhances multimedia capabilities and ap- 
plicability in communications technology by combining 
the properties of video, mobile phone and handwritten 
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text recognition in order to provide a versatile multime- 
dia communicator (Fig. 1). A terminal employing the 
method according to an embodiment of the invention en- 
ables persons with a hearing or speech defect to partic- 
ipate in a real-time video conference in which handwrit- 
ten notes may be stored, combined and quickly con- 
veyed by means of the invention to the participants to 
be further transmitted or stored as desired (Figs. 2 and 
3). 

[0013] Fig. 1 Illustrates the hierarchy of the develop- 
ment of the mobile phone, showing a conventional mo- 
bile station, or mobile phone, on the bottom of the "well". 
Conventionally, the characteristics of such a mobile 
phone include the functions required in the transmission 
and reception of voice. In the course of development the 
transmission and reception characteristics of the phone 
have become more diverse as various properties have 
been added in accordance with the figure. As a conse- 
quence the phone has become a means of communica- 
tion, a communicator, which can be made even more 
versatile and applicable when still more functions are 
added. 

[0014] Fig. 2 illustrates in the form of block diagram 
the operating principle of a mobile multimedia commu- 
nicator according to an embodiment of the invention. 
The user 200 writes text 201 on a touch-sensitive dis- 
play 202. By means of functions control 203 in the mo- 
bile multimedia communicator the user 200 converts 
213,214 the handwritten text 201 entered on the display 
202 into a pixel-based video image 204 and, if so de- 
sires, stores 227 it either in the working memory 207 or 
permanent memory 208. With the functions control 203, 
function 21 6 can be used to activate the handwriting rec- 
ognition algorithm 205 which reads 223 the pixel-based 
video image 204 and converts 22I4 the text 201 , 204 into 
a character system format 206. If the user 201 desires, 
the text 206 may be stored 225 by means of the multi- 
media functions control and timing 203 in the working 
memory 207 and further stored 226 in the permanent 
memory 208. Using the multimedia functions control 
and timing 203 the user 201 may pick 213 desired por- 
tions on the display 202 and input 214, 217, 219, 221 
them by means of functions to be used in the commu- 
nicator. In addition, by means of element 203 the user 
may fetch 232, through functions 215,218, 220 and 222, 
material onto the display 202 and receive 229 from a 
network 210 a signal that can be processed by functions 
214, 216, 217, 219 and 221 for the user and displayed 
232 on the display 202. By means of element 203 the 
user may choose components from the display 202, pix- 
el-based video image 204, character system based text 
206, working memory 207 or from the permanent mem- 
ory 208 to be transmitted 228 to the network 209. In that 
case the mobile station's read, display and transmission 
functions may be carried out in parallel to the extent and 
for the portions defined by the user. 
[0015] Fig. 3 illustrates the functions of the multimedia 
function control and timing 203 according to the inven- 



tion from the point of view of the user interface. A broken 
line defines the mobile station and the functions required 
of its user interface. The user 300 may use the mobile 
station 301 in conventional manner by receiving voice 

5 307 through the loudspeaker and/or transmitting voice 
308 through the microphone. In addition the user may 
read 309 a video signal on the display or transmit 31 0 
a video signal through the mobile station 301 1 say from 
a camera. Additionally, the display may be used for writ- 

10 ing in 311 graphic information when the text is in hand- 
writing. On the keypad of the mobile station it is possible 
to enter 312 text in ASCII format to be displayed 31 3 on 
the mobile station's display. In addition, it is possible to 
select an external information source 305, e.g. at the 

is end of a wire connection 314, for information input. The 
mobile station 301 also comprises a memory 306 for 
storing the material in question and placing it at the dis- 
posal of the user 300. The user 300 may by means of 
the mobile station 301 transmit, receive and/or present 

20 in accordance with Fig. 3 audio A, video V, text T or 
graphics G to the Internet or a corresponding network 
314 for the other participants 302, 303 and 304 in the 
call. 

[0016] Of the embodiments of the invention Fig. 4 

25 shows a mobile equipment 401 with which it is possible 
to detect 403 and recognize by means of a special text 
recognition element 41 2 text written on the display 402. 
The mobile station also includes system control 405 with 
which the user of the equipment may control 424 the 

30 multimedia system 406 and the modem 407 of the 
equipment 401 as well as the transmission 408, 413, 
425b and reception 409, 414, 425a functions. A video 
camera 411 may be connected to the mobile station so 
that the signal of said video camera is stored e.g. in 

35 MPEG-4 format on a video unit41 7. Usingfunction 423b 
it is possible to send 425b, 408 the video signal or por- 
tions of it via a mux/demux element 415 forward e.g. to 
a network or to other parts of the equipment such as the 
audio unit 41 8, handwriting recognition element 41 2, da- 

40 ta protocol block 41 2b and display 402. The mux/demux 
element 41 5 enables the userto control through the sys- 
tem control 405 and multimedia system 406 signals 
coming from the different elements 417, 418, 412, 415 
and apparatuses 411 , 410, 402, 407 and combine the 

45 signals in a desired manner and to distribute them to 
other elements or apparatuses as well as to be sent to 
a network to be used as video conference material, for 
example. 

[0017] The storage 212 of handwritten text 201 and 
50 the conversion of the text 201 , 204 into alphanumeric 
letters '(including characters specific to different cultural 
contexts as well as Cyrillic and Greek letters) can be 
performed 215, 217 in the manner chosen by the user 
through the functions control and timing 203. The text 
55 may be written 211 , 311 and/or stored during and/or be- 
fore a video conference to be distributed in video and/ 
or character format to selectable parties 301 , 302, 303, 
304 participating in the video call (Fig. 3) whereby the 
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